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SYSTEM AND METHOD FOR EVALUATING A PERSON'S INFORMATION 

TECHNOLOGY SKILLS 



CROSS REFERENCE TO RELATED APPLICATIONS 

This application is related to co-pending U.S. Patent Application No. 

(attorney docket no. 23415-012), filed on Febraary 01, 2002, and U.S. Patent 

Application No. (attorney docket no. 23415-013), filed on Febraary 26, 2002, 

both of which are assigned to the same assignee as that of the present invention and are 
incorporated herein by this reference. 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

The present invention is related to the field of evaluating a person's information 
technology (IT) skills. 

2. Discussion of the Background 

Today's businesses are extremely reliant on information technology (e.g., 
computers and other processing equipment) for many tasks. Consequently, when a 
business experiences a problem with a critical IT component (e.g., directory services, E- 
mail servers, web servers, or other IT components), the business may be adversely 
affected. Accordingly, businesses seek to hire one or more employees who have an 
expertise in certain computer-related areas to make sure that the business' IT systems 
run smoothly. 

Most employers do not have the means for determining whether a potential 
employee has the necessary computer skills for a particular job. Consequently, these 
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employers rely on IT certification organizations who certify individuals that possess a 
minimum amount of IT skill. That is, an employer may not hire a potential employee 
unless the potential employee has been certified for a particular skill by a well known 
certification organization. 

Accordingly, these certification organizations must develop tests for 
determining whether an individual should be certified. That is, whether the individual 
not only has the requisite IT knowledge, but also the ability to apply the knowledge to 
real world problems. 

Unfortunately, conventional IT certification tests may perform well at evaluating 
a person's IT knowledge, but are inadequate in other respects, such as when it comes to 
evaluating the person's IT skills (i.e., the person's knowledge as well as how well the 
person can apply their knowledge to solving real world problems). Other drawbacks 
exist. 

SUMMARY OF THE INVENTION 

The present invention aims to provide an IT skills evaluation system and method 
for evaluating an individual's IT skills, such as, for example, the individual's ability to 
solve a particular computer problem, load new software onto a computer, configure 
existing software on a computer, configure a computer's hardware devices, etc. 
According to one aspect of the invention, the individual is required to perform one or 
more practical exercises. A practical exercise is an evaluation device for evaluating not 
only an individual's knowledge, but also the individual's ability to apply his/her 
knowledge. According to one embodiment, one or more virtual machines are associated 
with each practical exercise, and the test taker uses the virtual machines in performing 
the exercise. Other implementations may be used. 

According to one aspect of the invention, a skills evaluation system includes a 
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testing computer on which at least the following software is installed: a virtual machine 
platform for allowing virtual machines to run on the testing computer and a practical 
skills evaluator for evaluating practical skills, for example, by presenting exams to a test 
taker and scoring the exam^ An exam is made up of one or more exam items. An exam 
item may include a question (e.g., a multiple choice question, essay question or other 
question) that the test taker is asked to answer. Additionally, an exam item may include 
a practical exercise that the test taker is asked to perform. Advantageously, a practical 
exercise includes a scenario and one or more tasks that the user is asked to complete 
based on the scenario. 

If a test taker takes an exam that includes a practical exercise, then, when the 
practical exercise is presented to the test taker, the practical skills evaluator (a) displays 
to the test taker information that describes the exercise's scenario and tasks that the test 
taker must complete to successfully perform the exercise, and (b) launches on the 
testing computer the one or more virtual machines that are associated with the exercise. 
The virtual machines are pre-configured so that they will have the necessary utilities 
and provide the necessary functions for the test taker to complete tiie tasks. Preferably, 
the virtual machines are placed into an initial, predetemiined state after being laimched. 
For example, a suspended state file may be created for each of the virtual machines so 
that, after a virtual machine is launched, the virtual machine is placed automatically into 
the initial predetermined state. 

After the virtual machines are laxmched, the test taker performs the tasks, which 
typically require the test taker to change the state of one or more of the virtual machines 
by, for example, installing a piece of software on one of the virtual machines, 
configuring an existing piece of software installed on the virtual machine, configuring a 
hardware device "connected'' to the virtual machine, modifying a system configuration 
file, or making some other change to the virtual machine. 

After completion of the practical exercise (e.g., after the test taker indicates that 
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he/she has completed the exercise or after a predetermined amount of time has elapsed 
from whence the exercise was presented to the test taker), the practical skills evaluator 
examines the state of the virtual machines (i.e., the changes the test taker made to the 
virtual machine) to determine whether the test taker completed the required tasks. The 
practical skills evaluator may also examine the state of the virtual machines to 
determine whether the test taker modified one or more of the virtual machines in a way 
that was not required. 

The above and other features and advantages of the present invention, as well as 
the structure and operation of preferred embodiments of the present invention, are 
described in detail below with reference to the accompanying drawings. 

BRIEF DESCRPTION OF THE DRAWINGS 

The accompanying drawings, which are incorporated herein and form part of the 
specification, illustrate various embodiments of the present invention and, together with 
the description, further serve to explain the principles of the invention and to enable a 
person skilled in the pertinent art to make and use the invention. In the drawings, like 
reference numbers indicate identical or functionally similar elements. Additionally, the 
left-most digit(s) of a reference number identifies the drawing in which the reference 
nxmiber first appears. 

FIG. 1 is a fimctional block diagram of an IT skills evaluation system, 
according to one embodiment, for evaluating a user's IT skills. 

FIG. 2 is a flow chart illustrating a process, according to one embodiment, that 
is performed by the IT skills evaluator. 

FIG. 3 is a fimctional block diagram of one embodiment of the IT skills 
evaluator. 
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FIG. 4 is a flow chart illustrating a process performed by a test driver. 

FIG. 5 is a flow chart illustrating a process performed by a practical skills 
testing module. 

FIG. 6 illustrates a virtual machine displayed within a testing window provided 
by the test driver. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

While the present invention may be embodied in many different forms, there is 
described herein in detail an illustrative embodiment with the understanding that the 
present disclosure is to be considered as an example of the principles of the invention 
and is not intended to Umit the invention to the illustrated embodiment. 

FIG. 1 is a functional block diagram of a system 100, according to an 
embodiment of the present invention, for evaluating user lOl's IT skills. System 100 
includes a testing computer system 102 that includes: a processor 104 for executing 
software, a storage medium 106 (e.g., hard disk, optical disk, or other storage medium) 
for storing data, software, and/or other information; a display device 108 for displaying 
infomiation to a user 101, and an input device 110 (e.g., keyboard, mouse, voice 
recognition device, and/or other input device) that enables system 102 to receive input 
from user 101. 

Computer system 102 includes a virtual machine platfomi (VMP) 112, which 
comprises virtual machine software for allowing multiple virtual machines to run 
simultaneously on computer system 102. Each virtual machine may execute the same 
or a different operating system. For example, one virtual machine may be configured to 
run the UNIX operating system, whereas another may be configured to run the 
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Windows NT® operating system. Computer system 102 further comprises an IT skills 
evaluator (ITSE) 1 14 for evaluating a test taker's (e.g., user lOl's) IT skills. Preferably, 
ITSE 114 comprises software for presenting exams to the test taker and scoring the 
exam. Both VMP 1 12 and ITSE 1 14 may be stored on storage medium 106 (or 
elsewhere) and may be executed by processor 104. 

When processor 104 executes software 112 and 1 14, computer system 102 is 
operable to: (a) present a practical exercise to user 101, which includes launching the 
one or more virtual machines that are associated with the practical exercise so that the 
user can perform the exercise using the virtual machines; (b) wait for completion of the 
exercise (e.g., for the user to complete the exercise, for a time-out to occur, or other 
completion event); and (c) evaluate how well user 101 completed the practical exercise. 
A practical exercise may include a scenario and one or more tasks that the user is asked 
to perform based on the scenario. 

In one embodiment, system 102 presents a practical exercise to user 101 by 
displaying information (e.g., text, graphics, sound, and/or other information) to the user 
that describes the practical exercise's scenario and tasks that user 101 must perform to 
successfiilly complete the exercise. In one embodiment, system 102 evaluates how well 
user 101 completed the practical exercise by examining the state into which the user put 
the virtual machines and comparing the state of the virtual machines to a benchmark 
state. In this way, system 102 determines whether user 101 completed the required 
tasks and whether user 101 modified a virtual machine in way that was not required. 

FIG. 2 is a flow chart illustrating a process 200 that is implemented by ITSE 114 
according to one embodiment. Process 200 may begin in step 202, where ITSE 114 
reads a test file 190. Test file 190 specifies an exam. An exam includes one or more 
exam items. An exam item may be a question, a practical exercise, or other exam item. 
A question may be a multiple choice question, an essay question or other question that 
user 101 is asked to answer. A practical exercise is an exercise that user 101 is asked to 
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perform. A practical exercise is associated with one or more virtual machines. Each of 
the one or more virtual machines is associated with a set of one or more virtual machine 
files 172, In one embodiment, virtual machines files 172 include, at the least, a disk 
image file (a.k.a., "virtual disk file"). Preferably, a suspended state file and/or other 
files are also included in file set 172. The disk image file stores the virtual machine's 
operating system, applications, data files, etc. 

Li step 204, ITSE 1 14 selects an exam item specified in test file 190 (or user 101 
selects the exam item). In step 206, ITSE 1 14 determines whether the selected exam 
item is associated with a practical exercise. If it is associated with a practical exercise, 
then control passes to step 210, otherwise control passes to step 208. 

In step 208, ITSE 1 14 presents the item (e.g., exam question) to user 101 and 
waits for user 101 to provide a response. In step 209, ITSE 114 evaluates the response 
(e.g., ITSE 1 14 determines whether user 101 correctly answered the question). After 
step 209, control passes to step 222. 

In step 210, ITSE 114 determines the virtual machines tiiat are associated with 
the practical exercise. In one embodiment, a database 170 is employed to associate 
practical exercises with virtual machine identifiers that identify the virtual machines 
with which the practical exercises are associated. In this embodiment, ITSE 114 uses 
database 170 to determine the virtual machines that are associated with the practical 
exercise. 

In step 212, ITSE 1 14 launches the one or more virtual machines determined in 
step 210. The virtual machines are pre-configured so that they will have the necessary 
software, utilities, etc. and provide the necessary fianctions for user 101 to complete the 
tasks required by the practical exercise. Advantageously, a suspended state file may be 
created for each of the virtual machines so that, after a virtual machine is launched, the 
virtual machine is placed automatically into a particular state defined by the virtual 
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machine's suspended state file. In this way, user 101 can be presented with a well 
controlled testing environment. For example, in a practical exercise that describes a 
scenario in which a computer is unable to access the Intemet and the task required of 
user 101 is to fix this problem, then a suspended state file associated with one or the 
exercise's virtual machines may be pre-configured so that when the virtual machine is 
presented to user 101 the virtual machine's IP routing table has incorrect information. 
In this way, the virtual machine emulates the "problem" described in the scenario, and 
user lOTs IT sills are evaluated by seeing if user 101 can modify the virtual machine's 
routing table to fix the problem. 

In step 214, ITSE 1 14 displays to user 101 information that is associated with 
the practical exercise. This information may be retrieved fi*om database 170 where it is 
stored and associated with the practical exercise. The information describes the 
practical exercise's scenario and the tasks that user 101 must perform to successfully 
complete the exercise. At this point, user 101 performs the tasks, which typically 
require user 101 to delete, create, and/or modify one or more particular files stored 
within on one or more of the virtual machines' disk image files (i.e., configure a piece 
of software installed on a virtual machine or a hardware device "'connected" to a virtual 
machine). 

In step 216, ITSE 1 14 sets a timer to expire after X seconds, minutes, or hours 
etc., where X can be determined from a configuration file or in other ways. In step 218, 
ITSE 1 14 waits for the timer to expire or for user 101 to indicate that he/she is finished 
with the practical exercise. After step 218, control may pass to step 220. 

In step 220, ITSE 1 14 evaluates how well user 101 performed the exercise and 
determines a "score" for the exercise (e.g., gives user 101a number of points based on 
how well user 101 performed the exercise or simply indicates that user 101 either 
passed or failed the exercise). In one embodiment, ITSE 114 evaluates how well user 
101 performed the exercise by, for example, examining the state of the virtual machines 
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(i.e., the changes the test taker made to the virtual machine) and comparing the state of 
the virtual machines to a benchmark. This enables ITSE 114 to determine whether user 
101 completed the required tasks (i.e., made the appropriate changes to the virtual 
machines) and whether user 101 made a change to a virtual machine that was not 
required. 

As an example, a task may require user 101 to delete, create, and/or modify one 
or more particular files stored within one of the virtual machine's disk image file. Thus, 
in evaluating whether user 101 completed the task successfully, ITSE 114 may examine 
the virtual machine's disk image file or other file within file set 172 to determine 
whether user 101 deleted, created, and/or modified the one or more particular files. 

Additionally, if user 101 should have modified a particular file in a particular 
way, the particular file is examined to see if user 101 modified the file appropriately. 
Also, ITSE 1 14 may examine the virtual machine's disk image or other file to 
determine which files user 101 deleted, created, and/or modified to determine if user 
101 deleted, created, and/or modified a file that should not have been deleted, created, 
or modified. Poiuts may be given to user 101 if user 101 deleted, created, and/or 
modified the appropriate files in the appropriate manner, and points may be deducted if 
user 101 deleted, created, and/or modified a file that was not required to be deleted, 
created, or modified. 

In one embodiment, database 170 may store file identifiers that identify the one 
or more particular files and associate these file identifiers with the practical exercise. 
Thus, ITSE 114 may retrieve fi-om database 170 the file identifiers that identify the files 
that rrSE 1 14 should examine in evaluating how well user 101 performed the practical 
exercise. Additionally, database 170 may store file identifiers that identify benchmark 
files associated with the practical exercise. In one embodiment, ITSE 114 compares the 
files that user 101 should have modified or created to a benchmark file. By comparing 
the files that user 101 should have modified or created to the benchmark files, ITSE 1 14 



Attorney Docket No,: 23415-014 
Novell Ref, No, IDR-552 



can determine whether user 101 made the appropriate changes to the files. 

After step 220, control passes to step 222. In step 222, ITSE 1 14 determines 
whether user 101 has finished the exam specified by test file 190. If user 101 has 
finished the exam, control passes to step 224, otherwise control proceeds back to step 
204. In step 224, ITSE 1 14 may grade the exam and display the results of the exam to 
user 101. 

In an altemative embodiment, ITSE 114 does not evaluate how well user 101 
performed the practical exercises. In this embodiment, the virtual machine files 172 are 
provided (e.g., electronically transmitted) to an evaluation center. The evaluation center 
may include computers that have the software for examining the files 172 to evaluate 
how well the user performed the practical exercises. Altematively, a person can 
manually examine the files 172 to evaluate how well the user performed the practical 
exercises. 

FICj. 3 is a block diagram of a one possible embodiment of ITSE 114. This 
embodiment is shown for the purpose of illustration, not limitation. One skilled in the 
art will recognize that there are a wide variety of ways to implement skills evaluator 
114. As shown if FIG. 3, ITSE 1 14 includes a conventional test driver 302 and a 
practical skills testing module (PSTM) 304, which may include one or more software 
modules. Test driver 302 may be obtained firom NCS Pearson, Inc., which provides the 
VUE™ computer-based test delivery service, or firom Prometric, Inc. of Baltimore, MD. 
Other Test drivers may also be used. 

FIG. 4 is a flow chart illustrating a process 400, according to one embodiment, 
that is performed by test driver 302. Process 400 may begin in step 402, where driver 
302 reads test file 190. Test file 190 specifies an exam having one or more exam items. 
In step 404, driver 302 selects an exam item specified in test file 190. In step 406, 
driver 302 determines whether the selected exam item corresponds to a practical 
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exercise. If it does correspond to a practical exercise control passes to step 410, 
otherwise control passes to step 408. 

In step 408, driver presents the exam item to user 101 and waits for user 101 to 
provide a response. In step 409, driver 302 evaluates the response (e.g., driver 302 
determines whether user 101 correctly answered the question). After step 409, control 
passes to step 414. 

In step 410, driver 302 provides to PSTM 304 an identifier that identifies the 
exam item selected in step 404. In step 412, driver waits for a response from PSTM 
304. The response may include information that indicates how well user 101 performed 
the practical exercise corresponding to the selected exam item. After step 412, control 
passes to step 414. 

In step 414, driver 302 determines whether user 101 has finished the exam. If 
user 101 has finished the exam, control passes to step 416, otherwise control proceeds 
back to step 404. In step 416, driver 302 may grade the exam and display the results of 
the exam to user 101. 

FIG. 5 is a flow chart illustrating a process 500, according to one embodiment, 
that is performed by PSTM 304. Process 500 may begin in step 502, where PSTM 304 
receives from driver 302 an identifier that identifies an exam item. The exam item 
identifies a practical exercise, which is associated with one or more virtual machines. 

In step 504, PSTM 304 determines the virtual machines that are associated with 
the identified practical exercise. In one embodiment, a database 170 is employed to 
associate exam item identifiers with virtual machine identifiers that identify the virtual 
machines with which the identified practical exercise is associated. In this embodiment, 
after PSTM 304 receives the exam item identifier from driver 302, PSTM 304 submits 
to database 170 a query that includes the exam item identifier and that requests database 
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170 to return the virtual machine identifiers that are associated with the exam item 
identifier. 

In step 506, PSTM 304 launches the one or more virtual machines determined in 
step 504. The virtual machines are pre-configured so that they will have the necessary 
utilities and provide the necessary functions for user 101 to complete the tasks required 
by the practical exercise. Advantageously, a suspended state file may be created for 
each of the virtual machines so that, after a virtual machine is launched, the virtual 
machine is placed automatically into a particular state defined by the virtual machine's 
suspended state file. 

Preferably, the virtual machines are presented to user 101 in a testing window 
602 provided by driver 302 (see FIG. 6). That is, the user interface generated by a 
virtual machine's operating system is displayed to user 101 so that user 101 may 
interact with the virtual machine's operating system, applications, and data files. In the 
embodiment shown in FIG. 6, each virtual machine is presented to user 101 in an easy 
to use tabbed interface. That is, window 602 includes a tab 604 for each virtual 
machine that was launched in step 504. In the example shown in FIG. 6, there are three 
tabs (tab 604(a), tab 604(b), and tab 604(c)). Thus, three virtual machines (e.g., VMl, 
VM2, and VM3) where launched in step 504. When user 101 wants to use VM3, user 
selects tab 604(c). After user 101 selects tab 604(c), VM3 is presented to user 101. 
Similarly, when user 101 wants to use VMl or VM2, user 101 selects tab 604(a) or 
604(b), respectively. 

In step 508, PSTM 304 displays to user 101 information that is associated with 
the identified practical exercise. This information may be retrieved firom database 170 
where it is stored and associated with the exam item identifier. The information 
describes the practical exercise's scenario and the tasks that user 101 must perform to 
successfiiUy complete the exercise. At this point, user 101 performs the tasks, which 
typically require user 101 to delete, create, and/or modify one or more particular files 
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stored on one or more of the virtual machines' disk image files (i.e., configure a piece 
of software installed on a virtual machine or a hardware device "connected" to a virtual 
machine). 

In step 510, PSTM 304 sets a timer to expire after X seconds, minutes, or hours 
etc., where X can be determined from a configuration file. In step 512, PSTM 304, 
waits for the timer to expire or for user 101 to indicate that he/she is finished with the 
practical exercise. User 101 may indicate that he/she is finished with the practical 
exercise by selecting "Next" button 690 (see FIG 6). After step 512, control may pass 
to step 514, 

In step 514, PSTM 304 evaluates how well user 101 performed the exercise and 
determines a score for the exercise (i.e., gives user 101 a number of points based on 
how well user 101 performed the exercise). In step 516, PSTM 304 provides to driver 
302 information concerning how well user 101 performed the exercise including user 
lOl's score for the exercise. 

While the processes illustrated herein may be described as a series of 
consecutive steps, none of these processes are limited to any particular order of the 
described steps. Additionally, it should be imderstood that the various illustrative 
embodiments of the present invention described above have been presented by way of 
example only, and not limitation. Thus, the breadth and scope of the present invention 
should not be limited by any of the above-described exemplary embodiments, but 
should be defined only in accordance with the following claims and their equivalents. 
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